Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

statically link libpq (fixes #213) #234

Closed
wants to merge 4 commits into from
Closed

Conversation

davepacheco
Copy link
Collaborator

@davepacheco davepacheco commented Sep 11, 2021

This was #233. If this works out, this will obviate #225 and finally resolve #213.

The Docker and Linux builds both fail because those distros don't ship libpgcommon.a and libpgport.a. Maybe we can build this separately like we do for Clickhouse and CockroachDB and have CI use those.

@davepacheco davepacheco marked this pull request as draft September 11, 2021 01:56
@davepacheco
Copy link
Collaborator Author

The reason this change doesn't work in the Docker image build and Linux CI is that they don't have a package installed that delivers libpgcommon.a or libpgport.a. The Docker image appears to be based on Debian either bullseye or sid, where these static archives are delivered by postgresql-server-dev-13. CI is based on ubuntu 18.04, which (not surprisingly) delivers it via a package of the same name. One problem is that this package delivers the libraries into /usr/lib/postgresql/13/lib. That's not where pg_config --libdir points -- it points to /usr/lib/x86_64-linux-gnu. And further, /usr/lib/postgresql/13/lib does not contain libpq.a, which is also needed (and PostgreSQL delivers them all three of these to the same directory). So to use this, we would need to specify an additional -L flag during the build, specifically for these environments and hardcoded to that path. They don't seem configurable in the usual ways (pg_config or pkg-config). @luqmana suggests setting LIBRARY_PATH to include them, which seems promising.

Basically, I think the ideas are:

(1) Modify the Docker image / CI environments to include the archives from these packages and make sure the build can find them. (What's described above.)
(2) Build our own PostgreSQL tarball containing these archives (similar to what we do for CockroachDB and Clickhouse) and use them in CI and the Docker image build.
(3) Modify pq-sys to do what openssl-sys does and have an option that downloads the PostgreSQL source, builds it, then statically links to that. This is sort of a way to do (2), but it would result in multiple copies if there were multiple components using pq-sys in the same deliverable.
(4) Transition the console to use Helios (presumably we'll want to do this eventually anyway but I don't know how much of the workflow this breaks), implement illumos CI, and drop Linux CI. FWIW I have found having dropshot and steno build on other platforms at least somewhat useful for smoking out non-portable stuff. I can't remember good examples, but I do feel like it caught some unnecessary dependencies.

@davepacheco
Copy link
Collaborator Author

In 3498daf, I did modify the Dockerfile to include the package with the static archives, but the Docker build now fails because of missing ldap and gss symbols. For reference, on this system, libpq.so has these dependencies:

# ldd $(pg_config --libdir)/libpq.so
	linux-vdso.so.1 (0x00007ffe4005b000)
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f9fff500000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f9fff20c000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f9fff1b9000)
	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f9fff163000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9fff141000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9ffef7c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9ffef74000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f9ffee9a000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f9ffee6a000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f9ffee64000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f9ffee55000)
	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f9ffee44000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f9ffee28000)
	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f9ffee0b000)
	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f9ffec0b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9fff5f0000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f9ffec04000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f9ffead0000)
	libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f9ffeaad000)
	libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f9ffe92b000)
	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f9ffe915000)
	libnettle.so.8 => /usr/lib/x86_64-linux-gnu/libnettle.so.8 (0x00007f9ffe8cd000)
	libhogweed.so.6 => /usr/lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007f9ffe884000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f9ffe803000)
	libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007f9ffe7f5000)

I didn't run into this on illumos. That's because this Linux build of libpq is built with a lot more options:

root@77cbf19c24d3:/usr/src/omicron# pg_config
BINDIR = /usr/lib/postgresql/13/bin
DOCDIR = /usr/share/doc/postgresql-doc-13
HTMLDIR = /usr/share/doc/postgresql-doc-13
INCLUDEDIR = /usr/include/postgresql
PKGINCLUDEDIR = /usr/include/postgresql
INCLUDEDIR-SERVER = /usr/include/postgresql/13/server
LIBDIR = /usr/lib/x86_64-linux-gnu
PKGLIBDIR = /usr/lib/postgresql/13/lib
LOCALEDIR = /usr/share/locale
MANDIR = /usr/share/postgresql/13/man
SHAREDIR = /usr/share/postgresql/13
SYSCONFDIR = /etc/postgresql-common
PGXS = /usr/lib/postgresql/13/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE =  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' '--with-icu' '--with-tcl' '--with-perl' '--with-python' '--with-pam' '--with-openssl' '--with-libxml' '--with-libxslt' 'PYTHON=/usr/bin/python3' '--mandir=/usr/share/postgresql/13/man' '--docdir=/usr/share/doc/postgresql-doc-13' '--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share/' '--datadir=/usr/share/postgresql/13' '--bindir=/usr/lib/postgresql/13/bin' '--libdir=/usr/lib/x86_64-linux-gnu/' '--libexecdir=/usr/lib/postgresql/' '--includedir=/usr/include/postgresql/' '--with-extra-version= (Debian 13.3-1)' '--enable-nls' '--enable-thread-safety' '--enable-tap-tests' '--enable-debug' '--enable-dtrace' '--disable-rpath' '--with-uuid=e2fs' '--with-gnu-ld' '--with-pgport=5432' '--with-system-tzdata=/usr/share/zoneinfo' '--with-llvm' 'LLVM_CONFIG=/usr/bin/llvm-config-11' 'CLANG=/usr/bin/clang-11' '--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' 'PROVE=/usr/bin/prove' 'TAR=/bin/tar' 'XSLTPROC=xsltproc --nonet' 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now' '--with-gssapi' '--with-ldap' 'build_alias=x86_64-linux-gnu' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security'
CC = gcc
CPPFLAGS = -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,-z,relro -Wl,-z,now -L/usr/lib/llvm-11/lib -Wl,--as-needed
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lpthread -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -ldl -lm 
VERSION = PostgreSQL 13.3 (Debian 13.3-1)

compared with the illumos one:

dap@epsilon3 omicron $ pg_config
BINDIR = /opt/ooce/pgsql-13/bin
DOCDIR = /opt/ooce/pgsql-13/share/doc
HTMLDIR = /opt/ooce/pgsql-13/share/doc
INCLUDEDIR = /opt/ooce/pgsql-13/include
PKGINCLUDEDIR = /opt/ooce/pgsql-13/include
INCLUDEDIR-SERVER = /opt/ooce/pgsql-13/include/server
LIBDIR = /opt/ooce/pgsql-13/lib/amd64
PKGLIBDIR = /opt/ooce/pgsql-13/lib/amd64
LOCALEDIR = /opt/ooce/pgsql-13/share/locale
MANDIR = /opt/ooce/pgsql-13/share/man
SHAREDIR = /opt/ooce/pgsql-13/share
SYSCONFDIR = /etc/opt/ooce/pgsql-13
PGXS = /opt/ooce/pgsql-13/lib/amd64/pgxs/src/makefiles/pgxs.mk
CONFIGURE =  '--includedir=/opt/ooce/pgsql-13/include' '--bindir=/opt/ooce/pgsql-13/bin/amd64' '--sbindir=/opt/ooce/pgsql-13/sbin/amd64' '--libdir=/opt/ooce/pgsql-13/lib/amd64' '--libexecdir=/opt/ooce/pgsql-13/libexec/amd64' '--bindir=/opt/ooce/pgsql-13/bin' '--enable-dtrace' 'DTRACEFLAGS=-64' '--prefix=/opt/ooce/pgsql-13' '--sysconfdir=/etc/opt/ooce/pgsql-13' '--localstatedir=/var/opt/ooce/pgsql-13' '--enable-thread-safety' '--with-openssl' '--with-libxml' '--with-xslt' '--with-readline' '--without-systemd' '--with-system-tzdata=/usr/share/lib/zoneinfo' 'CC=gcc' 'CFLAGS= -O2  -fno-aggressive-loop-optimizations -O3 -m64' 'LDFLAGS= -m64' 'CPPFLAGS= ' 'CXX=g++' 'CXXFLAGS= -O2  -fno-aggressive-loop-optimizations -m64' 'PKG_CONFIG_PATH=/opt/ooce/lib/amd64/pkgconfig'
CC = gcc
CPPFLAGS = -I/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation  -O2  -fno-aggressive-loop-optimizations -O3 -m64
CFLAGS_SL = -fPIC
LDFLAGS = -m64 -L/usr/lib/amd64 -Wl,-R'/opt/ooce/pgsql-13/lib/amd64'
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lxml2 -lssl -lcrypto -lz -lreadline -ltermcap -lnsl -lsocket -lm 
VERSION = PostgreSQL 13.0

Since we're linking statically, we have to add these dependencies ourselves. Now, we could modify pq-sys to emit "-l" options for these additional libraries too. But I believe some have different names on illumos, so this gets messy, and it feels increasingly like throwaway work. We'd be making the build system match the (contrived) execution environment when we actually have full control over the execution environment.

I looked briefly into how openssl-sys works when building from source. This really happens through openssl-src, which looks like this:
https://github.com/alexcrichton/openssl-src-rs/blob/7e9d7e9dabbdeee4a51037a7298bc32171a9dd03/src/lib.rs

It's not awful, but it's also not trivial. It's essentially exposing a Rust library interface around configure (with its various options) and make -- by executing the underlying shell commands.

@davepacheco
Copy link
Collaborator Author

In light of those issues, I went back to the dynamic linking approach -- see #225 (comment).

@davepacheco davepacheco deleted the issue-213-static branch March 13, 2022 03:11
@davepacheco davepacheco restored the issue-213-static branch March 13, 2022 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

build fails due to missing libpq
1 participant